Model registration system and method

ABSTRACT

System for registering a coordinate system associated with a pre-acquired model of an object with a reference coordinate system. The system includes a portable unit which includes a display, a tracking system for tracking the portable unit and a processor. The processor is coupled with the portable unit and with the tracking system. The processor determines the position and orientation of the portable unit. The processor further determines the position of at least one marker located on the object according to at least one of, a tracked pointer and respective position related information. The processor further displays registration related information on the display. At least one of the registration related information and the display location of the registration related information are related to the position and orientation of the portable unit. The position of the at least one marker, in the coordinate system associated with the pre-acquired model, is predetermined.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of U.S. patentapplication Ser. No. 15/531,685, filed on May 30, 2017, issued as U.S.Pat. No. 10,022,065 issued on Jul. 17, 2018, which is the U.S. NationalPhase Application under 35 U.S.C. § 371 of International Application No.PCT/IL2015/051160, filed Nov. 29, 2015, which claims priority to IsraeliApplication No. 236003, filed on Nov. 30, 2014, the entire disclosure ofeach of which is hereby incorporated by reference.

FIELD OF THE DISCLOSED TECHNIQUE

The disclosed technique relates to tracking systems in general, and tosystem and methods for registering a model of an object with a referencecoordinate system associated with a tracking system, in particular.

BACKGROUND OF THE DISCLOSED TECHNIQUE

Registering the coordinate system associated with an image of thepatient with the coordinate system associated with a medical trackingsystem enables the display of intraoperative information, (e.g., arepresentation of a medical tool, navigational information) on the imageof a body part of interest of a patient, at the respective positions andorientations thereof. Thus, the user may see such intraoperativeinformation along with the patient body part of interest.

U.S. Patent Application Publication U.S. 2011/0098553 to Robbins et aldirects to an automatic registration of a Magnetic Resonance (MR) imagewith an image guidance system. The registration is achieved by placingMR visible markers at known positions relative to markers visible in acamera tracking system. The markers are fixed to a common fixture whichis attached to a head clamp together with a reference marker (employedwhen the markers are covered or removed). The tracking system includes acamera with a detection array for detecting visible light and aprocessor arranged to analyze the output from the array. Each object tobe detected carries a single marker with a pattern of contrasted areasof light and dark intersecting at a specific single feature pointthereon with an array around the specific location. This enables theprocessor to detect an angle of rotation of the pattern and todistinguish each marker from the other markers.

U.S. Patent Application Publication 2012/0078236 to Schoepp, directs toa method for automatically registering the coordinate system associatedwith a navigation system with a coordinate system associated with a scanimage. Initially, a camera assembly of a navigation system, whichincludes fiducial markers, is fixedly attached to the patient (e.g.,with an adhesive). Thereafter, a scan image of the patient with thecamera is acquired. Scan image includes the camera with the fiducialmarkers. The registration module automatically recognizes and identifiesthe fiducial markers visible in the scan image and determines theposition of the camera assembly therefrom (i.e., the position of thefiducial markers with respect to the camera coordinate system and to thefocal geometry of the camera are known). The registration moduleautomatically registers the camera space with respect to the position ofthe patient in the scan image by identifying the position of the cameracoordinate system within the scan image. Upon automatic registration ofthe camera, the tracking of a surgical tool is immediately availablethrough the known relationships between the surgical tool, the cameracoordinate system, the scan image coordinate system.

SUMMARY OF THE PRESENT DISCLOSED TECHNIQUE

It is an object of the disclosed technique to provide a novel method andsystem for registering a model of an object with a reference coordinatesystem associated with a tracking system, in particular

In accordance with the disclosed technique, there is thus provided asystem for registering a coordinate system associated with a model of anobject with a reference coordinate system associated with the object.The system includes a portable unit, a tracking system and a processor.The portable unit includes a display. The processor is coupled with theportable unit and with the tracking system. The tracking system tracksthe portable unit in the reference coordinate system. The processordetermines the position and orientation of the portable unit in thereference coordinate system. The processor further determines theposition of at least one marker located on the object in the referencecoordinate system according to at least one of, a tracked pointer andrespective position related information. The processor also displaysregistration related information on the display, at least one of theregistration related information and the display location of theregistration related information being related to the position andorientation of the portable unit in the reference coordinate system.

In accordance with another aspect of the disclosed technique, there isthus provided method for displaying registration related informationcomprising the procedures of determining the position of markers in acoordinate system associated with a model of an object, the markersbeing located on the object and determining the position and orientationof a portable unit in a reference coordinate system, the portable unitincluding a display. The method further includes the procedures ofdetermining the position of at least three of the markers in a referencecoordinate system and registering the coordinate system associated witha model of the object with the reference coordinate system according tothe respective determined positions of the at least three of themarkers, in both coordinate systems. The method also includes theprocedure of displaying registration related information on the display,at least one of the registration related information and the displaylocation of the registration related information being related to theposition and orientation of the portable unit in the referencecoordinate system.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed technique will be understood and appreciated more fullyfrom the following detailed description taken in conjunction with thedrawings in which:

FIGS. 1A, 1B and 1C are schematic illustrations of an exemplary methodfor determining the location of fiducial markers located on an object,in accordance with an embodiment of the disclosed technique;

FIG. 2 is a schematic illustration of an exemplary optical trackingsystem for registering a coordinate system associated with a model of apatient body part with a coordinate system associated with a medicaltracking system, in accordance with another embodiment of the disclosedtechnique;

FIG. 3 is a schematic illustration of an exemplary electro-magnetictracking system employed for registering a model coordinate system witha reference coordinate system, constructed and operative in accordancewith a further embodiment of the disclosed technique;

FIG. 4 is a schematic illustration of an optical tracking system whichtracks the location of the portable unit in a reference coordinatesystem, constructed and operative in accordance with another embodimentof the disclosed technique;

FIG. 5 is a schematic illustration of an optical tracking system, whichtracks the location of the portable unit in a reference coordinatesystem, constructed and operative in accordance with a furtherembodiment of the disclosed technique;

FIGS. 6A, 6B, 6C and 6D are schematic illustrations of an exemplaryregistration process where registration related information is displayedto the user, during the registration process, in accordance with anotherembodiment of the disclosed technique;

FIG. 7 is a schematic illustration of a method for displayingregistration related information to a user, in accordance with a furtherembodiment of the disclosed technique;

FIG. 8 is a schematic illustration of a method for registering a modelcoordinate system and a reference coordinate system in accordance withanother embodiment of the disclosed technique;

FIGS. 9A and 9B are schematic illustrations of an exemplary standardmarker;

FIGS. 9C-9E are schematic illustrations of an exemplary activeregistration marker, constructed and operative in accordance with afurther embodiment of the disclosed technique;

FIG. 10 is a schematic illustration of cross-sectional view of a passiveregistration marker, constructed and operative in accordance withanother embodiment of the disclosed technique; and

FIGS. 11A and 11B are schematic illustrations of two exemplary fiducialmarkers, which may be employed for both model acquisition andregistration in accordance with a further embodiment of the disclosedtechnique.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The disclosed technique overcomes the disadvantages of the prior art byproviding a novel system and method for registering a model of an objectwith a reference coordinate system associated with a tracking system.The tracking system may be an optical tracking system, anelectro-magnetic tracking system, an ultrasonic tracking system, anoptical Time-Of-Flight tracking system. According to the disclosedtechnique, the tracking system tracks the position and orientation of aportable unit in the reference coordinate system. The portable unitincludes an optical detection assembly (e.g., sensor array camera, aPosition Sensitive Device—PSD, a stereoscopic camera or a Time-Of-Flightcamera). Prior to the registration process a model of the object (e.g.,a 2D or a 3D image of the head of the patient) is determined.Furthermore, the locations of at least three markers (i.e., fiducials oranatomical landmarks) are determined in the coordinate system associatedwith the model.

During the registration process, in order to determine the location offiducial markers in the reference coordinate system, the portable unitis held at a distance from the object. The user moves the portable unitaround the object through at least one registration positions. Eachregistration position is associated with a respective viewing angle ofthe fiducial. For example when the optical detection assembly of theportable unit includes an optical detector (e.g., sensor array camera ora PSD), then, the number of registration positions is at least two. Whenthe optical detection assembly of the portable unit includes astereoscopic camera or a TOF camera, the number of registrationpositions is at least one. For each registration position, the trackingsystem determines the position and orientation (P&O) of the portableunit in the reference coordinate system. Substantially simultaneouslytherewith, for each registration position, the tracking systemdetermines position related information respective of each fiducialaccording to the acquired image of the fiducial. When the portable unitincludes an optical detector (e.g., Charged Coupled Device—CCD camera ora Complementary Metal Oxide Semiconductor—CMOS camera or a PSD), theposition related information includes a respective directions towardeach of the at least one fiducial marker located on the object. Eachdirection defines a line in the reference coordinate system. Theintersection of the at least two lines associated with each fiducial(i.e., a line for each registration position), defines the location ofthat fiducial in the reference coordinate system. When the portable unitincludes, for example, a stereoscopic camera or a TOF camera, theposition related information may be related directly to the position ofthe fiducial in the reference coordinate system (e.g., two directionsfrom the two detectors in the stereoscopic camera or pixel depthinformation from the TOF camera). Also, the location of the markers(i.e., either of the fiducial markers or of the anatomical landmarks)may be determined with a pointer which is tracked in the referencecoordinate system. Since the coordinates of the markers in thecoordinate system associated with the model are known, the system candetermine the correspondence between the location of the markers in thereferenced coordinate system and the location of the markers in themodel coordinate system. Thus, registration between the coordinatesystem associated with the model and the coordinate system associatedwith the tracking system is achieved. Furthermore, the portable unit mayinclude a display. Also, herein, the term ‘located marker’ refers to amarker that the position thereof in the reference coordinate system wasdetermined.

When the tracking system is an optical tracking system, the trackingsystem may exhibit an in-out configuration, an in-out-out-inconfiguration or an out-in configuration. In the in-out configuration,the portable unit includes at least one optical detector, and areference unit, which is at a fixed position and orientation relative tothe object being tracked, includes at least three light emitters. In theout-in configuration the portable unit includes at least three lightemitters, and a reference unit includes at least one optical detector.In the in-out-out-in configuration the optical tracking system includesat least two optical detectors, one located on the portable unit and theother is located on a reference unit. Further in the in-out-out-inconfiguration, at least one light emitter is located on one of theportable unit and the reference unit and at least two light emitters arelocated on the other one of the portable unit and the reference unit(i.e., a total of at least three light emitters are employed). In boththe in-out configuration and the in-out-out-in configuration, an opticaldetector may be located on the portable unit and employed for bothtracking and marker detection (i.e., during the registration process).

In a tracking system employed for registration according to thedisclosed technique, the position and orientation of the reference unitare fixed relative to a patient body part. For example, the referenceunit is directly fixed to the patient body part. According to anotherexample, the patient body part is fixed and the reference unit is alsofixed, thus the reference unit is at fixed position and orientationrelative to the patient body part without being attached thereto. Thedisclosed technique may also be employed in other augmented realityscenarios.

Initially, prior to the registration procedure, a model of the patientis determined. This model may be, for example, a two-dimensional orthree-dimensional image of a region of interest of the body of thepatient (e.g., X-ray image, computed tomography—CT image, MagneticResonance Imaging—MRI image, ultrasound image, Proton EmissionTomography—PET image and the like). The model may be acquiredpre-operatively or intra-operatively. The model includes representationsof the at least three markers, which are employed as location points ofreference during registration of the coordinate systems. As mentionedabove, these markers may be artificial markers (i.e., fiducials) whichare attached to the patient prior to the acquisition of the model andremain attached to the patient until and during the registrationprocedure and optionally during the medical procedure which follows.Alternatively or additionally the markers may be anatomical landmarkswhich are visible in the model (e.g., the nose bridge or the tragus inthe ear). The location coordinates of these markers in the modelcoordinate system are determined by employing image processingtechniques or by manual localization on the image (e.g., with the aid ofa cursor).

Thereafter, and prior to the medical procedure, the locations of themarkers in the reference coordinate system associated with the trackingsystem are determined. Reference is now made to FIGS. 1A, 1B and 1Cwhich are schematic illustrations of an exemplary method for determiningthe location of fiducial markers located on an object for the purpose ofregistering the coordinate system associated with a model of the object,with a coordinate system associated with a tracking system, generallyreferenced 100, in accordance with an embodiment of the disclosedtechnique. Tracking system 100 in FIGS. 1A, 1B and 1C is an opticaltracking system which exhibits an in-out-out-in configuration. System100 includes a reference unit 108 and a portable unit 101. Portable unit101 includes a moving optical detector 102 associated with two lightemitters 104 ₁ and 104 ₂. Reference unit 101 is located, for example, onthe head of a user 106. Reference unit 108 includes a reference opticaldetector 109 associated with a light emitter 110. Reference unit 108,and thus light emitter 110 and optical detector 109 are in a fixedposition and orientation relative to a body part of patient 112. InFIGS. 1A-1C reference unit 108 is located on the head of patient 112. Ingeneral, reference unit 108 may be fixed relative to the body part ofthe patient without being physically attached thereto. In other wordsreference unit 108 and the body part of patient 112 do not move one withrespect to the other. In the example set forth in FIGS. 1A-1C opticaldetector 109 is a sensor array camera or a PSD. Thus, at least tworegistration positions are required.

To register the coordinate system associated with the model, with thecoordinate system associated with the tracking system, the locations ofthe markers in the coordinate system associated with the tracking systemshould be determined. To that end, the tracking system determines thelocation of the markers in a reference coordinate system. Accordingly,with reference to FIG. 1A, user 106 views patient 112 from a firstregistration position. Moving optical detector 102 detects light emitter110 and markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄. Reference opticaldetector 109 detects light emitters 104 ₁ and 104 ₂. The processordetermines the relative position and orientation between moving opticaldetector 102 and reference optical detector 109 at this firstregistration position, and thus the relative position and orientationbetween portable unit 101 and reference unit 108 in reference coordinatesystem 116. Reference coordinate system 116 is associated with referenceunit 108. Furthermore, the processor determines a first direction frommoving optical detector 102 toward each of markers 114 ₁, 114 ₂, 114 ₃and 114 ₄, relative to moving optical detector 102, according to therepresentations of markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄ detected bymoving optical detector 102, as explained below.

With reference to FIG. 1B, user 106 views patient 112 from a secondregistration position. Moving optical detector 102 detects light emitter110 and markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄ from this secondregistration position and reference optical detector 108 detects lightemitters 104 ₁ and 104 ₂ again. The processor determines the relativeposition and orientation between first detector 102 and reference unit108 at this second registration position, and thus the relative positionand orientation between portable unit 101 and reference unit 108, inreference coordinate system 116. Furthermore, the processor determines asecond direction from moving optical detector 102 toward each of markers114 ₁, 114 ₂, 114 ₃ and 114 ₄, relative to moving optical detector 102according to the representations of markers 114 ₁, 114 ₂, 114 ₃ and 114₄ detected by moving optical detector 102.

With reference to FIG. 1C, user 104 views patient 112 from a thirdregistration position. Moving optical detector 102 detects yet lightemitter 110 and markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄ from this thirdregistration position and reference optical detector 108 detects lightemitters 104 ₁ and 104 ₂ yet again. The processor determines therelative position and orientation between first detector 102 andreference unit 108 at this third registration position, and thus therelative position and orientation between portable unit 101 andreference unit 108, in reference coordinate system 116. Furthermore, theprocessor determines a third direction from moving optical detector 102toward each of markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄, relative to movingoptical detector 102 according to the representations of markers 114 ₁,114 ₂, 114 ₃ and 114 ₄ detected by moving optical detector 102.

The processor determines the location of each of markers 114 ₁, 114 ₂,114 ₃ and 114 ₄ in reference coordinate system 116, according to thethree directions associated with each one of marker 114 ₁, 114 ₂, 114 ₃and 114 ₄. For example each direction defines a line in referencecoordinate system 116 and the intersection of these three lines,associated with each marker, defines the location of that marker inreference coordinate system 116. In practice, the three lines may notintersect due to measurement errors and noise. Thus, for example, thepoint in space which exhibits the minimum sum of distances from thethree lines is determined as the location of the marker. Alternatively,for example, each determined direction may be associated with a FigureOf Merit (FOM) and each direction is weighted according to the FOMthereof.

The above description in conjunction with FIGS. 1A-1C describedregistering the coordinate system associated with the model, with thecoordinate system associated with the tracking system by employing threedifferent registration positions. However, in general, two registrationpositions are sufficient to determine the position of the markers in thereference coordinate system. Nevertheless, in practice, more than tworegistration positions are employed. For example, the registrationsystem automatically selects a plurality of discreet points in time(e.g., according to how fast the user is moving), determines theposition and orientation of the user in those points in time anddetermines a direction for each identified fiducial as described above.It is also noted that the portable unit may include two opticaldetectors directed substantially toward the same Field Of View (e.g.,stereoscopic camera). Consequently, detecting a fiducial with each ofthe two detectors is sufficient from a single user position (i.e.,assuming the fiducials are detected substantially simultaneously). Then,the system may triangulate the detected fiducial in order to determinethe location thereof in the referenced coordinate system.

Furthermore, the above description in conjunction with FIGS. 1A-1Crelates to fiducial markers (i.e., at least one of the markers is apassive or an active fiducials as further explained below), and thefiducial emits light (i.e., the fiducial includes either a light sourceor a light reflector) which can be detected by the optical detector inaddition to being detected by the imaging machine, as further explainedbelow.

The location of all or some of the markers (i.e., either fiducialmarkers or anatomical landmarks) may also be determined by employing atracked pointer, as further explained below. For example, the userplaces the tip of the pointer on the marker and the tracking systemdetermines the location of the tip of the pointer in the referencecoordinate system (i.e., similar to as performed in manualregistration). It is noted that if only a tracked pointer is employed todetermine the location of the markers, than the portable unit need notinclude an optical detection assembly. Since the locations of themarkers in the model coordinate system are known, the system candetermine the correspondence between the location of the marker in thereferenced coordinate system and the location of the markers in themodel coordinate system. When a tracked pointer is employed, theportable unit does not need to move through registration positions asexplained above.

Reference is now made to FIG. 2, which is a schematic illustration of anexemplary optical tracking system, generally reference 200, forregistering a coordinate system associated with a model of a patientbody part with a coordinate system associated with a medical trackingsystem, in accordance with another embodiment of the disclosedtechnique. System 200 may further be employed for tracking a medicaltool in a reference coordinate system. The tool may be superimposed on amodel of a patient 226. System 200 includes a first optical detector202, a second optical detector 204 and a reference unit 210. Referenceunit 210 further includes reference light emitters 212 ₁, 212 ₂ and 212₃. System 200 further includes a processor 214, a database 216 and adisplay such as HMD 218. HMD 218 includes a visor 220. HMD 218 may alsobe in the form of near-eye-display. HMD 218 and first optical detector202 define the portable unit. HMD 218 may also be replaced with aconventional screen (e.g., a hand-held tablet computer).

Processor 214 is coupled with database 216, first optical detector 202,HMD 218, second optical detector 204. When light emitters 206 ₁ and 206₂, or reference light emitters 212 ₁, 212 ₂ and 212 ₃ are LEDs,processor 214 is optionally coupled therewith. HMD 218 along with firstoptical detector 202 and light emitters 206 ₁ and 206 ₂ is donned by aphysician 224. Second optical detector 204 is attached to medical tool222. Reference unit 210, along with reference light emitters 212 ₁, 212₂ and 212 ₃ are all attached to a patient 226 body location (e.g., thehead, the spine, the femur), or fixed relative thereto. Patient 226 islying on treatment bed 228. In FIG. 2, the patient 226 body location isthe head of patient 226. System 200 is associated with a referencecoordinate system 230 which, in the system 200 is also the coordinatesystem associated with reference unit 210. In FIG. 2, the portable unitand reference unit 210 exhibit an in-out configuration. Furthermore, HMD218 is associated with a respective coordinate system 234. Also,markers, such as markers 232 ₁, 232 ₂ and 232 ₃, may be attached topatient 226. Although only three markers are depicted in FIG. 2, ingeneral, similar to as described in FIGS. 1A-1C, more than three markersmay be employed. Furthermore, at least one of markers 232 ₁, 232 ₂ and232 ₃ is a fiducial marker. Also, the remaining ones of markers 232 ₁,232 ₂ and 232 ₃ may be anatomical landmarks.

Processor 214 may be integrated within HMD 218 or attached to the user(e.g., with the aid of a belt or in the user's pocket). Medical tool 222is, for example, a pointer employed for determining the location of themarkers employed for registration. Medical tool 222 may also be anultrasound imager, a medical knife, a catheter guide, a laparoscope, anendoscope, a medical stylus or any other tool used by a physician 224during a procedure conducted on a patient 226. Also, the term coupledherein relates to either coupled by wire or wirelessly coupled.

In general, system 200 may be employed for registering the coordinatesystems associated with a model of patient 226 with reference coordinatesystem 230 as well as for tracking medical tool 222. Similar to asdescribed above, prior to registration, a model of the patient isdetermined which includes markers, such as marker 232 ₁, 232 ₃ and 232₃. Markers 232 ₁, 232 ₃ and 232 ₃ are employed as location points ofreference during registration procedure and the location coordinates ofthese markers, in the model coordinate system are determined (i.e.,employing image processing techniques or by manual localization on themodel). This model, along with the location coordinates of the markersis then stored in database 216.

Thereafter, physician 224 moves through at least two registrationpositions. For each registration position, first optical detector 202detects markers 232 ₁, 232 ₂ and 232 ₃ and light emitters 212 ₁, 212 ₂and 212 ₃. For each registration position, processor 214 determines theposition and orientation of HMD 218 (i.e., in reference coordinatesystem 230), according to the detected directions of light emitters 212₁, 212 ₂ and 212 ₃ and the known locations of light emitters 212 ₁, 212₂ and 212 ₃ on reference unit 210 (e.g., these locations are stored indatabase 216). Furthermore, for each registration position, processor214 determines a respective direction from HMD 218 toward each ofmarkers 232 ₁, 232 ₂ and 232 ₃. Processor 214 determines the location ofeach of markers 232 ₁, 232 ₂ and 232 ₃ according to the respectivedirections thereof at each registration position (e.g., the intersectionof the lines defined by each respective direction, define a locationpoint in reference coordinate system 230).

Also, physician 224 may employ a pointer to locate the markers (i.e.,either the fiducial markers or the anatomical landmarks). In such a casemedical tool 222 takes the form of a pointer. In order to determine thelocation of the markers, physician 224 places the tip of the pointer onthe markers. Second optical detector 204 also acquires an image of lightemitters 212 ₁, 212 ₂ and 212 ₃ and processor 214 determines thelocation of the pointer (i.e., of medical tool 222), and thus of themarker, in reference coordinate system 230. Similar to as mentionedabove, once processor 214 determines the position of the markers 232 ₁,232 ₂ and 232 ₃ (i.e., of the fiducials and the anatomical landmark) inreference coordinate system 230, processor 214 can register thecoordinate system associated with the model of the body part of patient226 with reference coordinate system 230.

When processor 214 determines at least an initial registration (e.g.,registration with a relatively large error) the coordinate systemassociated with the model of the body part of patient 226 with referencecoordinate system 230, processor 214 may display on visor 220registration related information as further explained below. Once thecoordinate system associated with the model of the body part of patient226 is registered with reference coordinate system 230, tracking system200 may be employed to track another medical tool (e.g., medical tool222 takes the form of a needle) in reference coordinate system 230.Furthermore, tracking system can superimpose a representation of such amedical tool on the model of patient 222. Also, according to thedetermined relative positions and orientations between medical tool 222,HMD 218 and patient 226, and the registration between the model ofpatient 226 and reference coordinate system 230, processor 214 mayrender the model of patient 226 in the correct perspective and providethe rendered model to HMD 218. Furthermore, navigational information(e.g., a mark representing a target location, a line representing thetrajectory and projected trajectory of the tool) associated with medicaltool 222, may be superimposed on the model. As a further example, whenmedical tool 222 is an ultrasound imager, system 200 be employed forpresenting data acquired by medical tool 222 at the location from whichthat data was acquired.

The light emitters described hereinabove in conjunction with FIGS. 1A-1Cand 2 may be either active light emitters (e.g., LEDs) or passive lightemitters which reflect either the ambient light or dedicated lightdirected thereat (e.g., the light from the LEDs located on the portableunit). The passive light emitters may be reflectors (e.g., reflectivespheres) or retro-reflectors which reflect light toward the directionfrom which it impinged thereon. The fiducial markers describedhereinabove in conjunction with FIGS. 1A-1C and 2 may also be passivefiducials or active fiducials. The passive fiducial also reflects thelight impinging thereon. The active fiducial includes a LED and abattery and is activated just before the registration process starts asfurther explained below in conjunction with FIGS. 9C-9E and 11A.

As mentioned above, the tracking system employed for registration mayalso be an electro-magnetic tracking system, which tracks the locationof the portable unit in a reference coordinate system. Reference is nowmade to FIG. 3, which is a schematic illustration of an exemplaryelectro-magnetic tracking system, generally referenced 250, employed forregistering a model coordinate system with a reference coordinatesystem, constructed and operative in accordance with a furtherembodiment of the disclosed technique. System 250 includes a referenceunit 252, a portable unit 254 and a processor 256. Reference unit 252includes a current generator 260 and magnetic field transmittingelements (e.g., coils) 262 ₁, 262 ₂ and 262 ₃. Portable unit 254includes an optical detection assembly 264 and magnetic field receivers266 ₁ and 266 ₂. Portable unit 254 also includes a display 268. Portableunit 254 may be embodied as an HMD similar to HMD 218 (FIG. 2) or a handheld unit (e.g., a hand-held tablet computer). Optical detectionassembly 264 is for example sensor array camera, a PSD, a stereoscopiccamera or a TOF camera.

Processor 256 is coupled with magnetic current generator 260, withoptical detection assembly 264, with magnetic field receivers 266 ₁ and266 ₂ and with display 268. System 250 aims to register the coordinatesystem associated with a model of object 258 with reference coordinatesystem 272. Object 258 includes at least three markers 270 ₁ 270 ₂ and270 ₃. At least one of markers 270 ₁ 270 ₂ and 270 ₃ is a fiducialmarker. In system 250, the position and orientation of reference unit252 are fixed relative to object 258. For example, reference unit 252 isdirectly fixed to object 258. Alternatively, object 258 is fixed andreference unit 258 is also fixed. Thus, reference unit 252 is at fixedposition and orientation relative to object 258 without being attachedthereto. Alternatively, at least two additional magnetic field receivers(not shown) are attached to object 258. Thus, processor 256 candetermine relative position and orientation between reference unit 252and object 258.

Similar to as described above in conjunction with FIGS. 1A-1C and 2, auser (not shown) moves portable unit 254 through at least tworegistration positions. For each registration position processor 256determines the position and orientation of portable unit 254 inreference coordinate system 272 according magnetic field transmitted bytransmitting elements 262 ₁, 262 ₂ and 262 ₃ and received by magneticfield receivers 266 ₁ and 266 ₂. For each registration position, opticaldetection assembly 264 acquires an image of the fiducial ones of markers270 ₁ 270 ₂ and 270 ₃. For each registration position processor 256determines a respective direction toward each of the fiducial ones ofmarkers 270 ₁ 270 ₂ and 270 ₃, relative to optical detection assembly264, according to the image acquired by optical detection assembly 264.Each direction defines a line in reference coordinate system 272 and theintersection of the three lines, associated with each marker, definesthe location of that marker in reference coordinate system. A user mayalso employ a tracked pointer (not shown) to determine the location ofmarkers 270 ₁ 270 ₂ and 270 ₃. Since the coordinates of the markers 270₁ 270 ₂ and 270 ₃ in the coordinate system associated with the model areknown, system 250 can determine the correspondence between the locationof markers 270 ₁ 270 ₂ and 270 ₃ in the referenced coordinate system 272and the location of the markers in the model coordinate system. Thus,registration between the model coordinate system and referencecoordinate system 272 is achieved. When processor 256 determines atleast an initial registration between the coordinate system associatedwith the model of object 258 with reference coordinate system 272,processor 256 may display on display 268 registration relatedinformation as further explained below.

Reference is now made to FIG. 4, which is a schematic illustration of anoptical tracking system, generally referenced 300, which tracks thelocation of the portable unit in a reference coordinate system,constructed and operative in accordance with another embodiment of thedisclosed technique. System 300 includes an optical tracking module 302,a portable unit 304 and a processor 306 which exhibits the out-inconfiguration. Portable unit 304 includes an optical detection assembly310 and at least three light emitters 314 ₁ 314 ₂ and 314. Portable unit304 also includes a display 312. In FIG. 4, light emitters 314 ₁ 314 ₂and 314 take the form of reflective spheres which reflect lightimpinging thereon. Optical detection assembly 310 is for example sensorarray camera, a PSD, a stereoscopic camera or a TOF camera.

Processor 306 is coupled with optical tracking module 302, with opticaldetection assembly 310 and with display 312. System 300 aims to registerthe coordinate system associated with a model of object 308 withreference coordinate system 318. Object 308 includes at least threemarkers 316 ₁ 316 ₂ and 316 ₃. At least one of markers 316 ₁ 316 ₂ and316 ₃ is a fiducial marker. In system 300, the position and orientationof reference unit optical tracking module 302 are fixed relative toobject 308.

Optical tracking module 302 may be embodied as stereoscopic camera(i.e., two cameras, directed toward substantially the same Field Of Viewand exhibiting a fixed and known relative position and orientationbetween the two cameras). Alternatively, optical tracking module 302 maybe embodied as a Time-Of-Flight (TOF) camera which includes a lightemitter which emits modulated light (e.g. continuous wave modulatedlight or pulsed modulated light) and an optical detector. When opticaltracking module 302 is embodied as a stereoscopic camera, processor 306determines the location of each one of light emitters 314 ₁ 314 ₂ and314 ₃ using triangulation. Thus, processor 306 can determine theposition and orientation of portable unit 304 in reference coordinatesystem 318. When optical tracking module 302 is embodied as a TOFcamera, each image includes the depth information of each pixel (i.e.,the distance between the TOF camera and the object being imaged) andeach pixel provides the direction from the TOF camera toward the objectbeing imaged. Thus, an image of light emitters 314 ₁ 314 ₂ and 314 ₃includes information relating to the location of these light emitters inreference coordinate system 318. Thus, processor 306 can determine theposition and orientation of portable unit 304 in reference coordinatesystem 318.

Similar to as described above in conjunction with FIGS. 1A-1C and 2, auser (not shown) moves portable unit 304 through at least tworegistration positions. For each registration position processor 306determines the position and orientation of portable unit 304 inreference coordinate system 318 according to the images acquire byoptical tracking module 302. For each registration position, opticaldetector 304 acquires an image of the fiducial ones of markers 316 ₁ 316₂ and 316 ₃. For each registration position, processor 306 determines arespective direction toward the fiducial ones of markers 316 ₁ 316 ₂ and316 ₃, relative to optical detection assembly 310, according to theimage acquired by optical detection assembly 310. Each direction definesa line in reference coordinate system 318 and the intersection of thetwo lines, associated with each marker, defines the location of thatmarker in reference coordinate system. The user may alternatively employa tracked pointer to determined location of markers 316 ₁ 316 ₂ and 316₃. Since the coordinates of the markers 316 ₁ 316 ₂ and 316 ₃ in thecoordinate system associated with the model are known, system 30 candetermine the correspondence between the location of markers 316 ₁ 316 ₂and 316 ₃ in the referenced coordinate system 318 and the location ofthe markers in the model coordinate system. Thus, registration betweenthe model coordinate system and reference coordinate system 318 isachieved. When processor 306 determines at least an initial registrationbetween the coordinate system associated with the model of object 308with reference coordinate system 318, processor 306 may display ondisplay 268 registration related information as further explained below.

Reference is now made to FIG. 5, which is a schematic illustration of anoptical tracking system, generally referenced 350, which tracks thelocation of the portable unit in a reference coordinate system,constructed and operative in accordance with a further embodiment of thedisclosed technique. System 350 includes a portable unit 352, areference unit 354 and a processor 356. Portable unit 352 includes anoptical tracking module 362 coupled with processor 356. Portable unit352 further includes a display also coupled with processor 356.Reference unit 354 includes at least three light emitters 360 ₁ 360 ₂and 360 ₃ and is attached to object 358. In the example brought forth inFIG. 5, light emitters 360 ₁ 360 ₂ and 360 ₃ are LEDs. Object 358includes three markers 366 ₁ 366 ₂ and 366 ₃, one of which is afiducial. Also, the relative position between reference unit 354 andobject 358 is fixed.

Similar to optical tracking module 302 (FIG. 4), optical tracking module362 may be embodied as a stereoscopic camera or a TOF camera. When theoptical tracking module includes a stereoscopic camera or a TOF camera,a single registration position is sufficient to determine the locationof markers 366 ₁, 366 ₂ and 366 ₃ in reference coordinate system 368(i.e., assuming all of the fiducial one of markers 366 ₁ 366 ₂ and 366 ₃are within the Field Of View of optical tracking module 362).

Accordingly, optical tracking module 362 acquires an image or images oflight emitters 360 ₁ 360 ₂ and 360 ₃ and processor 356 determines thelocation optical tracking unit 362 and consequently of portable unit 352in reference coordinate system 368. Also, optical tracking module 362acquires an image or images of the fiducial one of markers 366 ₁ 366 ₂and 366 ₃, and processor 356 determines the location of markers 366 ₁366 ₂ and 366 ₃ relative to optical tracking module 362. Since processor356 determined the location of optical tracking unit 362 in referencecoordinate system 368, processor 356 can determine the location of thefiducial ones of markers 366 ₁ 366 ₂ and 366 ₃ in reference coordinatesystem 368. The user may alternatively employ a tracked pointer (e.g.,tracked in a coordinate system associated with portable unit 352) todetermined location of markers 366 ₁ 366 ₂ and 366 ₃. Since thecoordinates of the markers 366 ₁ 366 ₂ and 366 ₃ in the coordinatesystem associated with the model are known, system 360 can determine thecorrespondence between the location of markers 366 ₁ 366 ₂ and 366 ₃ inthe referenced coordinate system 368 and the location of the markers inthe model coordinate system. Thus, registration between the modelcoordinate system and reference coordinate system 368 is achieved. Whenprocessor 356 determines at least an initial registration between thecoordinate system associated with the model of object 358 with referencecoordinate system 368, processor 356 may display on display 268registration related information as further explained below.

In the examples brought herein above in conjunction with FIGS. 1A-1C, 2,and 5, the optical detection assembly located on the portable unit isemployed for both tracking the portable unit and for registration.However, the portable unit may include two separate optical detectionassemblies, one employed for tracking the portable unit and the otheremployed for registration.

With respect to any of the tracking systems described hereinabove inconjunction with FIGS. 1A-1C, 2, 3, 4 and 5, during the registrationprocess, information relating to the registration process may bedisplayed to the user (i.e., on the respective display associated withany one of the tracking system described hereinabove in conjunction withFIGS. 1A-1C, 2, 3, 4 and 5). This registration related information maybe, for example, a marker identifier (e.g., a number, a character), anindication that a marker has been identified, an indication that amarker has been located, the error associated with the determinedlocation of the marker, a score indicating the quality of theregistration (e.g., the estimated error of the registration),instructions to the user and the like. For example, once the location ofa marker is determined, a marker indicator may be displayed to the user,for example, by superimposing the indicator (e.g., a circle, a square,an arrow and the like) on the marker, thus providing the user withinformation regarding the progress of the registration process. Eachkind of marker (i.e., either fiducial or anatomical landmark) may have acorresponding indicator (e.g., a circle for fiducials and a square foranatomical landmarks). When the positions of a sufficient number ofmarkers are determined (i.e., at least three when registering threedimensional coordinate systems) and registration is calculated, a scoreindicating the quality of the registration may be displayed to the user.The registration related information may further include user relatedinformation such as user selection or user guidance. For example, theuser may direct the tracking system whether the score is good enough orwhether to continue the registration process (e.g., by enabling thesystem to locate additional markers). For example, when the markers arelocated on both sides of the head, then the system may direct the userto physically look at the head of the patient from the other side toallow the system to identify additional markers. To improve the accuracyof the registration, the system may further guide the user to look atthe head of the patient from the other side, even if the registrationwas already successful using markers from only one side of the head ofthe patient. Once an initial registration is determined (e.g., may bewith a large error), the system may also direct the user (e.g., via thedisplay) to markers that location thereof has yet to be determined orthat the location thereof was determined with a large error. The systemmay also indicate to the user the error that each marker contributed tothe final calculation of the registration. The user may also discard theuse of specific markers in the calculation of the registration.Discarded markers may be indicated with a different indicator than themarkers that were employed for registration (e.g., discarded markersshall be marked with a red square). For example if the user suspectsthat certain markers may have moved since the preoperative image hasbeen acquired. Also, the surgeon may request that the registration berecalculated without using certain markers.

Reference is now made to FIGS. 6A, 6B, 6C and 6D, which are schematicillustrations of an exemplary registration process where registrationrelated information is displayed to the user, for example on a visor400, during the registration process, in accordance with anotherembodiment of the disclosed technique. The user observes patient 402lying on treatment bed 404. In the example set forth in FIGS. 6A-6D, areference unit 406 is at a fixed position and orientation relative tothe head of patient 402. Reference unit 406 may be any one of thereference units described above in conjunction with FIGS. 1A-1C, 2, 3, 4and 5. In the example brought forth in FIGS. 6A-6D, reference unit 406includes three LEDs 408 ₁, 408 ₂ and 408 ₃. Alternatively, referenceunit may include magnetic field transmitters or receivers as explainedabove. Furthermore, marker 401 ₁, 410 ₂, 410 ₃, 410 ₄, 410 ₅, 410 ₆ and410 ₇ are located on patient 402 (i.e., either fiducials or anatomicallandmarks or both).

With reference to FIG. 6A, the user is located at a first registrationposition. At this first registration position, the user views markers410 ₁, 410 ₂, 410 ₃ and 410 ₄. A registration system according to thedisclosed technique identifies markers 410 ₁, 410 ₂ 410 ₃ and 410 ₄(e.g., markers 410 ₁, 410 ₂, 410 ₃ and 410 ₄ are within the field ofview of an optical detector) and informs the user (e.g., by displayingtext on visor 400) that four markers have been identified. Furthermore,the system (e.g., any of the systems described hereinabove) instructsthe user to change the point of view thereof. It is noted that when astereoscopic camera or a TOF camera are employed with the portable unit,the system is also able to determine the location of markers 410 ₁, 410₂ 410 ₃ and 410 ₃ from a single registration position.

With reference to FIG. 6B, the user is located at a second registrationposition. At this second registration position, the user views markers410 ₂, 410 ₃, 410 ₄, 410 ₅ and 410 ₆. The registration system accordingto the disclosed technique further identifies markers 410 ₅ and 410 ₆(e.g., markers 410 ₂, 410 ₃, 410 ₄, 410 ₅ and 410 ₆ are within the fieldof view of an optical detector) and informs the user that 6 markers havebeen identified. Furthermore, the registration system determines thelocation of markers 410 ₂, 410 ₃ and 410 ₄ and displays respectivemarker indicators 412 ₂, 412 ₃ and 412 ₄ on display 400, superimposedover the respective marker thereof, as seen by the user through thetransparent visor. Since the system according to the disclosed techniquetracks a portable unit in the reference coordinate system, the systemcan determine the P&O of the display. Since the system also determinesthe location of the markers, the system can superimpose a markerindicator at the display location which is related to the position ofthe markers as seen on or through the display. In general, theregistration system displays the registration related information at adisplay location which corresponds to the position and orientation ofthe portable unit. The registration system may display the registrationrelated information at a display location which is related to theposition of the markers. In FIGS. 6B-6D, marker indicator 412 ₂, 412 ₃and 412 ₄ take the form of circles. The registration system alsoprovides the user with an indication of the error of the determinedlocation thereof. For example, the registration system determined theposition of marker 410 ₂ with an error of 0.9 mm, marker 410 ₃ with anerror of 0.3 mm and marker 410 ₄ is located with an error of 0.5 mm. Itis noted that the size, color or shape of the marker indicator may berelated to the error associated with the position of that marker. Forexample, the diameter of the circle is proportional to the location ofthe marker over which that circle is superimposed. Since three markershave been identified, the system can estimate the registration betweenthe reference coordinate system and the coordinate system associatedwith the model of the patient. However, this registration may be aninitial registration with a relatively large error (e.g., 2.5millimeters in the example set forth in FIG. 6B). Nevertheless, sincethe spatial relationship (i.e., the relative position) between themarkers is known, the system can instructs the user to move towardmarkers which are yet to be detected (i.e., in general, the registrationrelated information includes instructions to the user). In FIG. 6B, thesystem informs the user that 6 markers have been identified and 3located. Furthermore, the system displays on display 400 instructions tothe user to move to the left.

With reference to FIG. 6C, the user is located at a third registrationposition. At this third registration position, the user views markers410 ₄, 410 ₅ 410 ₆ and 414 ₇. The registration system according to thedisclosed technique further identifies marker 410 ₇ (e.g., markers 410₄, 410 ₅, 410 ₆ and 410 ₇ are within the field of view of an opticaldetector). The registration system according to the disclosed techniquedetermines the location of markers 410 ₅ and marker 410 ₆ and marksthese markers with a respective circle 412 ₅ and 412 ₆. The systemdetermined the position of marker 410 ₅ with an error of 0.4 mm, marker410 ₆ with an error of 0.5 mm. Furthermore, the system improved thelocation estimation of marker 410 ₄ and the location error associatedwith marker 410 ₄ is now 0.3 mm. The system was not able to determinethe location of markers 414 ₇ as well as of marker 410 ₁. The systemalso improved the registration error (e.g., 0.66 millimeters in theexample set forth in FIG. 6C). The system further instructs the user tomove to the left.

With reference to FIG. 6D, the registration system according to thedisclosed technique displays on display 400 a summary of theregistration process for the user and indicates that the registration iscomplete and the displays the registration error (i.e., the registrationrelated information includes, for example, a summary of the registrationprocess for the user and indicates that the registration is complete anddisplays the registration error). The system further displaysinformation relating to the markers employed for registration andvarious options for the user to choose from. In general, as explainedabove, the system displays registration related information to a user ata display location related to the position and orientation of theportable unit. It is noted that since the system determines the positionof the markers, the system may adjust the information displayed on thedisplay accordingly. For example, the marker indicators may be displayedat a display location corresponding to the position of the markers asseen on or through the display, while the registration error,instructions to the user and the like may be displayed at a differentselected location which does not interfere with the marker indicators.

Reference is now made to FIG. 7, which is a schematic illustration of amethod for displaying registration related information to a user, inaccordance with a further embodiment of the disclosed technique. Inprocedure 420, markers within the field of view of an optical detectorare identified. The markers are fixed to an object. These markers may befiducial markers or anatomical landmarks. With reference to FIGS. 6A-6D,markers 410 ₁, 410 ₂, 410 ₃, 410 ₄, 410 ₅, 410 ₆ and 410 ₇, which arewithin the field of view of an optical detector are identified.

In procedure 422, the positions of at least some of the identifiedmarkers, in a reference coordinate system, are determined. Furthermore,the position error of the identified markers is also determined. Withreference to FIGS. 6A-6D, a processor (not shown) determines theposition of at least some of markers 410 ₁, 410 ₂, 410 ₃, 410 ₄, 410 ₅,410 ₆ and 410 ₇ in reference coordinate system 414. When the positionsof at least three markers are identified, the method proceeds toprocedure 424. Otherwise, the method returns to procedure 420.

In procedure 424, the coordinate system associated with a model of theobject is registered with the reference coordinate system, according tothe respective positions of the at least three of the identified markersin both coordinate systems. Furthermore, the registration error isdetermined. With reference to FIGS. 6A-6D a processor registersreference coordinate system 414 with the coordinate system associatedwith a model of patient.

In procedure 426, registration related information is determined anddisplayed to the user. As mentioned above, registration relatedinformation may further include user related information such as userselection or user guidance. With reference to FIGS. 6A-6D, registrationrelated information is displayed on visor 400.

In procedure 428, the user is directed to move in a direction whereadditional markers would be within the field of view of the opticaldetection assembly. Since at least initial registration is determined,the location of all markers in the reference coordinate system can beestimated. Thus, the location of these markers relative to the locationof the portable unit can also be determined. It is noted that directingthe user in a direction where additional markers would be within thefield of view of the optical detection assembly is optional and mayoccur when the registration process is yet to be completed (e.g., whenthe registration error is above a threshold or the user selects tocontinue the registration process). With reference to FIG. 6B, the useris directed to move to the right in order to identify and locatedadditional markers. After procedure 428 the method returns to procedure420.

In general, there are three types of error estimations involved in theregistration process. The first is the error estimation (herein ‘typeone error estimation’) relates to the error of the position of a singlemarker in the reference coordinate system. This error results from theresidual error of the triangulation process (i.e., lines intersection),the angular difference between the lines and the location error of theportable unit. This error may be relatively large when the marker waspartially obscured from some direction, smudged by blood and the like,or when the angular difference between the directions associated withthe marker is relatively small. In such a case the user may beinstructed to move to another registration position so the marker may besampled from an additional direction. The error may also be large if theuser moved relatively fast while the marker was sampled (i.e., when thedirection from the portable unit toward the marker was determined). Suchan error may be detected automatically and the user may be instructed,for example, to move slower. The second type of error estimation foreach marker (herein ‘type two error estimation’) relates to the distancebetween the position of the markers in the registered model coordinatesystem (i.e. the image coordinate system after the rotation andtranslation onto the tracker coordinate system according to thecalculated registration) and the position of the marker in the referencecoordinate system. A specific marker may have been displaced between thetime the imaging was performed and the time the registration isperformed, but still be accurately located. In such a case, this markerwill exhibit a small estimated error of the first type and a largeestimated error of the second type and the system may discard itautomatically or recommend to the user to discard it manually.Consequently, the registration may be improved. The third type of errorestimation (herein ‘type three error estimation’) is the figure of meritof the registration calculation, which may be the average of the errorsof the second type for all the markers, or any other objective function(i.e., the objective of the registration calculation is to minimize thiserror). All of the above types of error estimations may be calculatedand displayed to the user (e.g., in millimeters).

Reference is now made to FIG. 8, which is a schematic illustration of amethod for registering a model coordinate system and a referencecoordinate system in accordance with another embodiment of the disclosedtechnique. In procedure 450, the position of each of the at least threemarkers is determined in a coordinate system associated with a model ofan object. At least one of the at least three markers is a fiducialmarker. When the model is, for example, an image, the location of themarkers may be determined by employing image processing techniques.Alternatively, the location of markers may be manually marked on ascreen. After procedure 450, the method proceeds to procedure 460.

In procedure 452 the position of at least one anatomical landmark isdetermined in the reference coordinate system, when at least oneanatomical landmark is employed as a marker. With reference to FIG. 2,when the at least part of the markers are anatomical landmarks,physician 224 employs a pointer. In such a case medical tool 222 takesthe form of a pointer. Physician 224 places the tip of the pointer onthe anatomical landmark. Processor 214 determines the location of thepointer (i.e., of medical tool 222), and thus of the marker, inreference coordinate system 230 as described above. After procedure 452,the method proceeds to procedure 460.

In procedure 454, for each of at least one registration position, theposition and orientation of a portable unit in a reference coordinatesystem is determined. The portable unit includes an optical detectionassembly. When the optical detection assembly is an optical detector(e.g., a sensor array camera or a PSD) then, the number of registrationpositions is at least two. When the optical detection assembly is astereoscopic camera or a TOF camera, the number of registrationpositions is at least one. With reference to FIGS. 1A-1C, a user 106moves moving optical detector 102 (i.e., which, as mentioned above,defined the portable unit together with light emitters 104 ₁ and 104 ₂)through at least two registration positions. Moving optical detector 102acquires at least one image of light emitter 110 and moving opticaldetector acquires at least one image of light emitters 104 ₁ and 104 ₂.A Processor (e.g., processor 214—FIG. 2) determines the position andorientation of the relative position between reference optical detectorand a moving optical detector is determined in reference coordinatesystem 116 according to the representations of light emitters 104 ₁, 104₂, and 110. With reference to 5, optical tracking module 362 may beembodied as either a TOF camera or a stereoscopic camera which acquireswhich acquires an image or images of light emitters 360 ₁ 360 ₂ and 360₃. Processor 356 determines the location of optical tracking unit 362,and consequently of portable unit 352, in reference coordinate system368.

In procedure 456, for each of the at least one registration position,location related information respective of each of the at least onefiducial that are within the field of view of the optical detectionassembly, is determined. When the portable unit includes an opticaldetector (e.g., a sensor array camera or a PSD), the position relatedinformation includes a respective directions toward each of the at leastone fiducial marker located on the object. When the portable unitincludes, for example, a stereoscopic camera or a TOF camera, theposition related information may be related directly to the position ofthe fiducial in the reference coordinate system (e.g., two directionsfrom the two detectors in the stereoscopic camera or pixel depthinformation from the TOF camera). With reference to FIGS. 1A-1C, whenmoving optical detector 102 acquires the image or images of lightemitter 110, moving optical detector 102 also acquires and image ofmarkers 114 ₁, 114 ₂, 114 ₃ and 114 ₄. For each registration position,the processor determines position related information of markers 114 ₁,114 ₂, 114 ₃ and 114 ₄, relative to moving optical detector 102,according to the image of markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄. Withreference to FIG. 5, optical racking module 362 With reference to 5,optical tracking module 362 may be embodied as either a TOF camera or astereoscopic camera, which acquires an image or images of markers 366 ₁366 ₂ and 366 ₃. Processor 356 determines the location of markers 366 ₁366 ₂ and 366 ₃ determines the location of the fiducial ones of markers366 ₁ 366 ₂ and 366 ₃ in reference coordinate system 368.

In Procedure 458, the position of each of the at least one fiducialmarker located on the object is determined in the reference coordinatesystem, according to the positions and orientations of the portable unitin each of at least two registration positions and the respectiveposition related information of each of the at least one fiducialmarker. For example each direction defines a line in the referencecoordinate system. The intersection of the at least two directionsassociated with each fiducial defines the location of that fiducial inthe reference coordinate system. As mentioned above, in practice theselines may not intersect. In such a case, the point exhibiting theminimum distance to each of the lies is determined as the location ofthe marker. With Reference to FIGS. 1A-1C and 2, a processor (e.g.,processor 214—FIG. 2), determines the position of each of the at leastthree markers (e.g., markers 114 ₁, 114 ₂, 114 ₃ and 114 ₄ in FIG. 2 or232 ₁, 232 ₂, 233 ₃ in FIG. 2) in reference coordinate system (e.g.,referenced coordinate system 116 in FIG. 1 or reference coordinatesystem 230 in FIG. 2).

In procedure 460, the coordinate system associated with the model of theobject is registered with the reference coordinate system, according tothe respective positions of at least three of the at least threemarkers, in both coordinate systems. With Reference to 2, processor 214registers the coordinate system associated with the model of the objectwith reference coordinate system 230, according to the respectivepositions of the markers in both coordinate systems.

The description herein above relates to an automatic registrationprocess with an augmented reality environment, where the registrationsystem displays registration related information overlaid on thedisplay, at a display location which corresponds to the position andorientation of the portable unit and the location of the markers in areference coordinate system. In general, each one of the displaysdescribed above may be hand held or head mounted) or part of anyportable unit in general (e.g. attached to a moveable arm). For example,a video see-through portable unit includes a tablet computer and acamera. A video see-through portable unit may alternatively include anHMD with a non-transparent near-eye display and a video camera. In avideo see-through portable unit the video from the camera is augmentedand displayed to the user in the display. When an optical trackingsystem is employed for tracking a video see-through portable unit, thecamera employed for tracking and for the video see-through may be oneand the same. An optical see-through portable unit includes, forexample, a tablet computer with a transparent display, or a projectorand a half-silvered mirror attached to a movable arm. An opticalsee-through portable unit may alternatively include an HMD with avisor-projected display or a transparent near-eye display.

The descriptions herein above exemplified the registration process withthe user moving through at least two different registration positions.However, in practice, when the location of the markers is determinedwith the aid of the portable unit, the user may move the portable unitwithout constraints around the patient, while maintaining the patientwithin the FOV of the optical detector of the portable unit. The opticaldetector detects the markers during the motion of the portable unit(e.g., acquires an image when an imaging sensor is employed). Thetracking system determines the position and orientation of the potableunit each time a marker is detected and determines the location of themarkers as described above, both at a relatively high frequency (e.g.,on the order tens of times per second).

Reference is now made to FIGS. 9A-9E. FIGS. 9A and 9B are schematicillustrations of an exemplary standard marker, generally referenced 500.A standard marker 500 is employed during model acquisition (e.g., duringCT or MRI imaging). FIGS. 9C-9E are schematic illustrations of anexemplary active registration marker, generally reference 510,constructed and operative in accordance with a further embodiment of thedisclosed technique, which may be attached to a standard marker 500.Active registration marker 510 is employed during registration. FIG. 9Ais a top view of a standard marker 500 and FIG. 9B is a cross sectionview of a standard marker 500. In the example brought forth herein, thestandard marker 500 is in the form of a ring which forms a cavity 508.Standard marker 500 includes a marker body 502, and a bottom sticker504. Bottom sticker 504 is employed for attaching marker 350 to thepatient. Marker body 502 is made of a material which may be detected inthe acquired model (e.g., a radio-opaque material for CT imaging).Marker 500 may also have a cover 506 that protects the marker fromdamage and is removed before the registration process.

As mentioned above, the markers described hereinabove in conjunctionwith FIGS. 1A-1C 2, 3, 4, 5 and 6A-6D, may be passive markers or activemarkers. A passive marker reflects the light impinging thereon. Anactive marker includes a LED and a battery and is activated just beforeinitiation of the registration process starts. With reference to FIG.9C, active registration marker 510 includes a housing 512, an LED 514, apower supply 516, a detachable isolator 518, a protrusion 520 and asticker 522. LED 514 is coupled with power supply 516. Detachableisolator 518 isolates LED 514 from power supply 516. In general, powersupply 516 takes the form of a battery. However, power supply 516 mayalso take to form of a preloaded capacitor. With reference to FIG. 9D,before active registration marker 510 is attached to standard marker500, sticker 522 is removed exposing an adhesive. Thereafter, protrusion520 is inserted into cavity 508 and housing 512 is fixedly attached tomarker body 502. With reference to FIG. 9E, once active registrationmarker 510 is attached to marker body 502, detachable isolator 518 isremoved thereby connecting LED 514 to power supply 516. Thus, LED 514starts to emit light.

As mentioned above, the registration marker may also be a passiveregistration marker. Such a passive registration marker may be areflector or a retro-reflector. Reference is now made to FIG. 10, whichis a schematic illustration of cross-sectional view of a passiveregistration marker, generally referenced 550, constructed and operativein accordance with another embodiment of the disclosed technique.Passive registration marker 550 is exemplified herein as a corner cuberetro-reflector. Passive registration marker includes a housing 552, acorner cube retro-reflector 554, a protrusion 558 and a sticker 560.Corner cube retro-reflector 554 includes three mirrors. Two mirrors 556₁ and 556 ₂, of the three mirrors included in a corner cube reflector554 are depicted in FIG. 8. Light impinging on corner cuberetro-reflector is reflected back toward the direction from which thatlight arrived. Similar to active registration marker 510 (FIGS. 7C-7E),passive registration marker may be fixedly attached to a standard markersuch as marker 500 (FIGS. 5A-5B), after the model acquisition processand before the registration process.

In general, the passive registration marker 550 is illuminated with theLED located on the portable unit (e.g., LEDS 104 ₁ and 104 ₂ of FIG. 1or LEDs 206 ₁ and 206 ₂ of FIG. 2). The optical detector located on theportable unit (e.g., optical detector 102 of FIG. 1 or optical detector202 of FIG. 2) acquires an image of the light reflected from passiveregistration marker 550. Thus, when passive registration marker 550 isembodied as a retro-reflector, it is important that the light emittersof the portable unit be located sufficiently close to the opticaldetector such that the light that is retro-reflected from passiveregistration 550 could be detected by the optical detector.

It is noted that the, according to the disclosed technique, a singlefiducial marker may be employed during both model acquisition andregistration. Reference is now made to FIGS. 11A and 11B, which areschematic illustrations of two exemplary fiducial markers, generallyreference 600 and 620 respectively, which may be employed for both modelacquisition and registration in accordance with a further embodiment ofthe disclosed technique. Fiducial marker 600 is an active fiducialmarker and fiducial marker 620 is a passive fiducial marker.

With reference to FIG. 11A, fiducial marker 600 includes a body 602which is made of a material which may be detected in the acquired model(e.g., a radio-opaque material for CT imaging), a sticker 604, an LED606 a power supply 608 (e.g., a battery or a capacitor) and a detachableisolator 610. LED 606 is coupled with power supply 608. Detachableisolator 610 isolates LED 606 from power supply 608. Bottom sticker 604is employed for attaching marker 600 to the patient. Thereafter, themodel of the patient is acquired. Prior to the registration process,detachable isolator 610 is removed thereby connecting LED 606 to powersupply 608. Thus, LED 606 starts to emit light.

With reference to FIG. 11B, fiducial marker 620 includes a body 622 ismade of a material which may be detected in the acquired model, asticker 624 and corner cube retro-reflector 626. Corner cuberetro-reflector 626 includes three mirrors. Two mirrors 628 ₁ and 628 ₂,of the three mirrors included in a corner cube reflector 626 aredepicted in FIG. 7B. Bottom sticker 624 is employed for attaching marker620 to the patient. Thereafter, the model of the patient is acquired.

It will be appreciated by persons skilled in the art that the disclosedtechnique is not limited to what has been particularly shown anddescribed hereinabove. Rather the scope of the disclosed technique isdefined only by the claims, which follow.

The invention claimed is:
 1. A system for registering a coordinatesystem associated with a pre-acquired model of an object with areference coordinate system, the system comprising: a portable unitincluding: a display; and an optical detection assembly for detecting atleast one representation of each of at least one marker located on saidobject; a tracking system for tracking said portable unit in saidreference coordinate system; and a processor, coupled with said portableunit and with said tracking system, said processor configured todetermine the position and orientation of said portable unit in saidreference coordinate system, said processor further configured todetermine at least position related information from said at least onerepresentation, said processor further configured to displayregistration related information on said display, at least one of saidregistration related information and the display location of saidregistration related information being related to the position andorientation of said portable unit in said reference coordinate system,wherein the position of said at least one marker in said coordinatesystem associated with said pre-acquired model is predetermined.
 2. Thesystem according to claim 1, wherein said registration relatedinformation includes at least one of: a marker indicator; a markeridentifier; an error associated with the determined location of amarker; a registration score; instructions to said user; or userselection options.
 3. The system according to claim 2, where said userinstructions include at least one of: instruction to change view point;or instruction to move in a specified direction.
 4. The system accordingto claim 2, wherein at least one of: the size of said marker indicator,the shape of said marker indicator, or the color of said markerindicator is related to at least one of type one error estimation ortype two error estimation associated with each marker.
 5. The systemaccording to claim 1, wherein when at least some of said markers arefiducial markers, said processor is further configured to identify atleast one of said fiducial markers being within the field of view ofsaid optical detection assembly, said optical detection assembly beingattached to said portable unit, wherein for each of at least oneregistration positions, said processor is further configured todetermine respective position related information of each of said atleast one marker being within the field of view of said opticaldetection assembly, and wherein said processor is further configured todetermine the position of each of said at least one marker in saidreference coordinate system according to the respective position andorientation of said portable unit in each of said at least oneregistration position and said position related information respectiveof each of said at least one marker.
 6. The system according to claim 5,wherein said position related information is at least one of: adirection from said portable unit toward said marker; at least twodirections from said portable unit toward said marker; or a directionand a distance from said portable unit toward said marker.
 7. The systemaccording to claim 5, wherein said processor is further configured todirect said user to move in a direction where additional markers wouldbe within the field of view of said optical detection assembly.
 8. Thesystem according to claim 5, wherein at least three light emitters areattached to said object, and wherein said processor is furtherconfigured to determine the position and orientation of said portableunit according to at least one representation of each of said at leastthree light emitters acquired by said optical detection assembly.
 9. Thesystem according to claim 5, wherein a second optical detection assemblyis attached to said object, wherein, at least one light emitter islocated on said object and at least one light emitter is located on saidportable unit, wherein, the total number of said light emitters is atleast three, and wherein said processor is further configured todetermine the position and orientation of said portable unit accordingto at least one representation of said at least one light emitterlocated on said object and acquired by said optical detection assemblyand at least one representation of said at least one light emitterlocated on said portable unit and acquired by said second opticaldetection assembly.
 10. The system according to claim 1, wherein saidoptical detection assembly is a three dimensional optical detectionassembly.
 11. The system according to claim 10, wherein said opticaldetection assembly is a stereoscopic camera, and wherein said positionrelated information is at least two directions from said portable unittoward said marker.
 12. The system according to claim 10, wherein saidoptical detection assembly is a time-of-flight camera, and wherein saidposition related information is a direction toward said marker and adistance between said portable unit and said marker.
 13. The systemaccording to claim 1, wherein, said optical detection assembly is oneof: a sensor array camera or a position sensitive device, and whereinsaid position related information is at least one direction from saidportable unit toward said marker.
 14. The system according to claim 1,wherein at least some of said markers are anatomical landmarks.
 15. Thesystem according to claim 14, wherein said processor is furtherconfigured to determine the position of at least some of said markersaccording to the position and orientation of a tracked pointer inresponse to determining that said pointer touches each of saidanatomical landmarks.
 16. The system according to claim 1, wherein saidportable unit is one of a head mounted display, a unit attached to amovable arm, or a hand held unit.
 17. The system according to claim 16,wherein said display is one of: optical see through display; or videosee through display.
 18. The system according to claim 1, wherein theoptical detection assembly comprises a position sensitive device, andwherein the representation comprises a direction of an arrival of lightincident.
 19. The system according to claim 1, wherein the opticaldetection assembly comprises an imaging sensor, and wherein therepresentation comprises an image.
 20. The system according to claim 19,wherein the optical detection assembly further comprises a positionsensitive device, and wherein the representation further comprises adirection of an arrival of light incident.
 21. A method for displayingregistration related information, the method comprising: determining theposition of markers in a coordinate system associated with apre-acquired model of an object, said markers being located on saidobject; determining the position and orientation of a portable unit in areference coordinate system, said portable unit including a display andan optical detection assembly for acquiring at least one representationof each of said markers; determining the position of at least three ofsaid markers in a reference coordinate system from said at least onerepresentation and said position and orientation of said portable unit;registering said coordinate system associated with said pre-acquiredmodel of the object with said reference coordinate system according tothe respective determined positions of said at least three of saidmarkers, in both coordinate systems; and displaying registration relatedinformation on said display, at least one of said registration relatedinformation and the display location of said registration relatedinformation being related to the position and orientation of saidportable unit in said reference coordinate system, wherein the method isperformed by at least one processor.
 22. The method according to claim21, wherein said registration related information includes at least oneof: a marker indicator; a marker identifier; an error estimationassociated with the determined location of a marker; a registrationerror estimation; instructions to a user; or user selection options. 23.The method according to claim 22, where said user instructions includeat least one of: instruction to change view point; or instruction tomove in a specified direction.
 24. The method according to claim 22,wherein at least one of the size of said marker indicator, the shape ofsaid marker indicator, or the color of said marker indicator is relatedto at least one of type one error estimation or type two errorestimation associated with each marker.
 25. The method according toclaim 21, where said determining the position of at least three of saidmarkers further includes determining the position error estimationassociated with each located marker, and wherein said registeringfurther includes determining the registration error estimation.
 26. Themethod according to claim 21, wherein when at least some of said markersare fiducial markers, said determining the position of at least three ofsaid markers includes, for at least one marker, the method furthercomprises: identifying said at least one marker within the field of viewof said optical detection assembly, said optical detection assemblybeing attached to said portable unit; for each of at least oneregistration position, determining respective position relatedinformation of each of said at least one marker being within the fieldof view of said optical detection assembly; and determining the positionof each of said at least one marker in said reference coordinate systemaccording to the respective position and orientation of said portableunit in each of said at least one registration position and saidposition related information respective of each of said at least onemarker.
 27. The method according to claim 26, further includingdirecting said user to move in a direction where additional markerswould be within the field of view of said optical detection assembly.28. The method according to claim 26, wherein said position relatedinformation is at least one of: a direction from said portable unittoward said marker; at least two directions from said portable unittoward said marker; or a direction and a distance from said portableunit toward said marker.
 29. The method according to claim 21, whereinat least some of said markers are anatomical landmarks.
 30. The methodaccording to claim 29, wherein said determining the position of at leastsome of said anatomical markers is performed with a tracked pointer.